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Experimental Research- (EIFER), an iliventory for appraising research 

"q'uality^ iis described- The' Elf ER was specifically designed to aid 
students' in introductory courses in educational and psychological 
re'sea-rch methods to evaluate published research reports. A survey 
conducted among specialists in educational research resulted in a 
list of 73 characteristics considered 'essential in experimental and 
quasi-experimental research- These characteristics wer^ organized 
into 6 categories: the research problem, review of related 

, literature, research design, data collection and analysis , . 
conclusions and generalizations, and ; sty\te and organization . The 
following psychometric/properties- of J the 'inventory, ^were determined: 
consistency, item-section correlation, analysis of variance, 
interrat.er and inttarater reliability, and stability of the. 
measurement , cf traits across raters jaii^. occasions . Reliability 
--coefficients -^^ere determined to be satisfactory. A copy of the 
instrument ' is included. ( Authcr/GDC)| ' . \y- . . . 
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Introduction/' - 

Research studies vary widely in quality" and published 
reports of them- ;diff er; significantly in value. . Therefore, an 
answer to' the, question, "How much/ confidence can be placed in 
the results reported in a study?/ is one of great' importance.* 
Although professional journals pf education devote considerable 
space to. reports of educational research, many readers have 
difficulty in evaluating- their wprth. Cn:e reason for' this 
difficulty may be the absence of agreed-upon criteria in terms 
of which, to maJce the judgments which are simple enough for a - 
statistically naive reader to apply. Although there have been 
many proposals for appraising published research reports in 
educaxion and psychology (Brooks, 1923; Davitz & Davitz, 1967; 
Parquhar. & Krumboltz, ,t Fox, 1958; Ingle cS: Gephart, 1966; 

Suydam, 1968, Wandir, /1 9^65; -Ward, .Hall, & Schramm, ,1:975) , 
almost all of them, however, have been ^ui¥e7complex and demand 
considerable sophistication on th^ part of their users ^ 

It- was the intenx of the res^earch reported here to produce 
a set of criteria which can be used to evaluate important 
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aspects of published research reports and yet which are simple 
enough to be understood and applied by neophytes who have had 
guided training in their use in beginning courses in methods of 
educational research. The development of these criteria, then, 
rests upon these premises: (1) essential characteristics of 
valid research can be' identified; (2) some of these are simple 
enough to be understood and.applied by students with limited 
backgrounds who are enrolled in a first course in educational 
research dethods; and (3; if these students study the es^ablj^shed 
"criteria carefully and havp guided practice in the appraisal 
of research' in terms o'f them, they should become more proficient 
in making sound ^judgments concerning the internal as well as 
external validity of the results reported in a specific study. 

.. ..■ , Purpose 

Ihe ob;jective of the study was to develop an inventory for ' 
appra,ising experimental and quasi-experimental' research "designed ■ 
specifically for introductory -research methods classes in 
education and psychology. She inventory should posses such 
essential characteristics a's content validity, internal consis- 
tency, and stability-of-response reliabilities . 

An .extensive review of literature had revealed that quite.., 
a few- of . appraisal forms had been developed for the use of 
evaluating reported educational research ('Hsu, 1975;.o They 
would generally appear in one of the three" forms: article- type ,. 
checklists and rating instruments. Regardless of the format 



in which the instruments were presented, they tended. to contain 
one. or. more of the flaws, identified below:. 

1. 2he -procedures of development were not described, 

2. A key or scales were not provided for judging the 
. criteria included. 

3. The specifications of the scale, If available, were 
either undefined or very ambiguous. 

. 4. No empirical data were generated to support the adequacy 

of the key or scales provided. 

•.5* Psychometric characteristics, of the instrument were 

either incomplete or not established ^t all. 

6. She type of research for which the particular instrument 

is intended was not speci^^iTed^ 

As a matter of fact, none.of the evaluation instruments - 

identified by the authors possessed the -three attributes which • 

-characterize the inventory developed in this study: (1) it is 
■ ■ / ■ ■ .... ■ ■ ■ 

intended for use in teaching the beginning students of research ^ 

methods the key points , in the apipraisal of experimental 

•pd.ucational research;, (2) such: essential psychometric information 

as the inter- rat.Jp ^and intrar-rater reliability 'indices was ' 
* ■ / ' " ■ ■ . ■ . • • 

established; and (3) its appropriate ' usefulness 

:^or instructional purposes was e?:.plored empirically through an 

experiment conducted in classes of research methods .at the . ^ 

'University of Georgia,' Athens, Georgia." • 



. procedures 

/ Ihe overall objective of this study was to develop an 
inventory of essential characteristics of experimental research. 
In-order to accomplish this task, several stages were formulated 
as shown in /figure 1 . • ' 
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.. Establishment of Content Validity 
2o ascertain the suitability of the evaluation instrument 
as an instructional aid to students of the beginning research 
methods classes, its content validity must be properly estab- 
lished, .5:he 'following steps were taken to achieve this goal: 
t • A survey was made of - texts on research methods and. of 
journal literature reporting the^construction and .use 
of instruments for fappraising educa'tlonal ' research. 
, 2. Based on .the sib:vey, a list was prepared of character- " 
istics. essential, to the execution and reporting of " 
experimental and quas'i-experimental^.research. \ 
3* The list; was -then sent to a hation-^Wide panel of 35 . , . 
. educational research specialists 'for their indeT)endent 
judgments as to the essentiality of each characteristic 
included. • , ' - . . . ' ■' 

4. Characteristic judged as essential by 17 or more of the 
21 specialists responding were retained. Based on" this 



criterion, 46 out of the 51 items identified in Step 2 ' 
above- were retained in their original forms or retained 
after revision. "" 

5. The list of retained '^characteristics was. again given a 
nation-wide panel of 40 educational research methods'" 
instructors listed in.-^t-he membership directory of- the 
American Educational Research Association Special 
■Interest Group: Prof essors ' of Educational Research. 
She panel was asked to independently appraise each'" 
characteristic as one appropriate for inclusion in- 
introductory research methods courses. 

6. Characteristics judged appropriate by 30 or. more of the 
34 who responded were included in the final version of 
the -inventory. In addition, comments or' suggestions • 
given by the members of the' panel were used as guidelin 
for rearranging or restructuring some of the statements 
if deemed necessary. . ' ' • 

\ ■ Sele<;tion of Foils " 
A set of structured, responses (foils) was needed for ' use . 
with each inventory statement of an essential characteristic, 
thereby enabling the student to appraise the research. in terms 
of that characteristic more precisely by selecting one- of the 
foils. ■ A review of "similar inventories and logical consider.--, 
ations reduced the types of options considered to only two: 
categorical versus continuous. Th^ categorical type was 
.equipped with a definition for each of the five responses 



listed while the continuous type had a five-point scale, 
ranging from 1 for "inadequate*' to 5 for .^'adequate", with 2,. / 
3, and 4 set in hetween undefined. 

' Per the selection of the key to he incorporated in the 
inventory, a pilot study was conducted in a. graduate class of 
an advanced educational psychology cours e. The students v/ere 
randomly divided into two groups, with one using the cat egorical 
key and the other the continuous type, to rate an article of 
experimental research in education. On the basis of the" 
empirical comparison made in the pilot study, one set of the* 
foils was selected for use with the evaluation instrument. 

Check on Internal Consistency and 
. i . ' Stahility-of-"Resi3ohse Reliabilities 

Since xhe inventory was developed primarily for the ^ 

evaluation of published reports of research in experimental or 

quasi-experimental design, a sample of research reports -of this 

nature was selected for use in the pilot study and in a subsequent 

study to ch^ck on the psychometric properties' of the instrument. 

Articles ya the American Educational Research Journal (AERJ) 

published in the most recent four years of 1970-73', inclusive, 

were surveyed to identify research articles of this specific 

design. Two were randomly selected , approved for use' by; AERJ , 

and ."blinded" as to author , and publication. Eight instrijictors 

with' experience in. teaching graduate level 'research methGlds at 

the Uni versity of Georgia were asked -^to use the inventory and 

appraise the two selected articles. V< ithout .forelmowled^e of 

the request, each was asked to reappraise the ■ two/ articles ' 



approximately "one moniih later. These appraisals provided 
data for Raters by Articles by (Traits oy Occations analyses of 
variance (ANOlTAs) and a portion of the data for the Kuder- 
Richardson formula 20 (K-R 20) reliability estimates (see 
Figure 2 for-thp lay-out of these AlfOVAs and reliability 
estimates}. Moreov,er, „ members of a gradiiate research s.eminar 

' Insert pigure 2 about Here • , . * ■ ' 

also appraised and re-appraised the two articles at about. the 
same time^ and under the same instructions as the instructors • 
Eleven of the - 1 3 graduate students completed twc /evaluations of 
each articles. Data from all of these .sets of apipraisals were 
used for a series, of estimates of the K-R' 20 reliability ' 
coefficients. - 

Check on the Use'fulness of the Inv entory 
The attributes contained in the inventory were essential 
characteristics, of experimental research. They were prepared 
specifically for use in educating beginning students of research 
methods. Hence.it was necessary; to ascertain its usefulness by 
conducting an experiment in; classes of research methods. 

In this experiment, the experimental group (S) consisted' 
of a random half of each of five introductory research methods 
classes. They (with a total of 58' Ss). were given copies of 
the inventory and of one of the two' articles. The other half 
from the same = five classes constituted the'control group (C) • • 



with a total of 55 subjiects. Q^liey vrere given copies of the 
same article but not the inventories* . Both groups ere asked 
to appraise each of the six "global^' aspects^ (corresponding to 
the six inventory sections) of the article. 'The hypothesis 
tested was that the proportions of inventory-users whose 
"global.'', a^^ with those of the eight instructors 

would be greater than the proportions of non- users who agreed. 

The data were obtained a few days before the end of the quarter, 

\! ' * 
Results 

I^'y^^tory C.ont ent 
In its final f orA.,^t he inventory , Evaluation Instrument fo r- 
Ibcperimental Research (EIPER), contains a list of 73 character- 
istics categorized ijito six major sections, each pertaining to 
a major aspect of experimental research - Research Problem, 
Relented Literature, Research Design, Data Collection and 
Analysis, Conclusions and G-eneralizations, and Style and 

Organization of the Report, At the end of each section is an 
* • 

it^m pertaining to- a ^'global" appraisal with respect to that 
particular section (see Appendix 1). 

^ ' Structure of the Foils ' 

Prom, th4 results of the pilot study, an empirical 
comparison was made of the two sets of the foils mentioned 
. earlier , on 'the basis of the ' variabilities of the amoraisals 
with each set, which were determined from the-sum of weighted 
absolute ^deviations from the modal, response , as well as of a 



survey of user's preference for the' keys. The empirical 
findings strongly favored the same set, the categorical type, 
which is listed below Calso see Appendix i ) t 

V: ihefarticie contains no information concerning the 
— ^attribute. ; , , / 

2: The information given clearly indicates that the 

■ attribute was .inappropriately handled^ 

3: The inforaation given suggests that t'hfe attribute may 

,have been inT)X"gT)erly .managed. 
4: The inform'^tion given indicates that the attribute was 

t)roT3efly m^aged. — 
5: -The information given .clearly indicates that the 

attribute was apprdioriat ely handled. , ■ 

Estimates of Selected Psychometric Characteristics . 
The primary psychometric measures estimated for EIPER 
included: measures of -internal consistency in terms of .KR-20 
indices, item-section Int ercorrelatlons ^- analysis of variance 
(AN07A) fpr each section of the inventory, interrater- 
reliabilities , ■ intrarater. reliabilitiesj, and. stabilities of; . ■ 
xiaeas^ares of traits across raters and occasions.. 

Measures of internal consistency . ! To test the -homogeneity 
of inventory items, an estimate of reliability was nroduced via 

■ ■ , ' - .: V . . .. . - ^ - ■ . 

KR-20 (r^^). Shese reliability coefficients are the average 
correlations obtained from ail possible split-half reliability 

/estimates. As shown in Table\ 1 , all nine estimated coefficients 

■ ■ ■ ' ■ ■ , \ ■ ■ o 

for total inventory score were^ ♦90, or above, six of the nine 
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for the set of "global" appraisals were .■80 or above, and of 
the 54 section KiEl-20s, 17 of the .33 with .30 or higher were 
.90 or above while only four were lower than .60. • 

Table. 2 contains- the, correlations between EIPER sections.' 
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OJhe correlation coef ficients . ranged from the lowest (r > . 28) J 
^letween Research Problem and Conclusions and Generalizations to 
the highest '(r =".69) between Rfesaarch Design and Data | Collection, 
and Analysis, with fflost of the pair^vise relationship of other 
sections; falling within the moderate range. This gave a- rather 
clear indication tha1i the rattribute^ contained in each section 
were not repeated of nor overlapped with those across other 
sections.- Such an indication, ^.n -turn, supported the accomplish- 
ment, of the major function by each EI'PER sedtion in measuring- 
different rather than similar or. identi^cal aspect of the ,■ 
reported researcli ar.ticle. \ A.'* ' . / 

Item-^ section intercorrelatioVis , ?or the purpose of; checking 
on zhe "goodness of fit" of each inventory item- in the particular 
section, a matrix of* it em- section co:ij:]r^elat ions was generated V 
(see 2able 3)^ Ideally, a specific item should ■ correlate, higher 
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with its ovm 'section than ' wixll'' others of the. inventory to 
■val!i.date' its "legitimate nesting" in that, particular s emotion. 
Evidently, except for sdme of the items in the Related Literature 
section, a greaf..fflajofi.ty of "the items in the other six EIPER ■ 
y sections correlated higher with the sectioh in'which they were 
. . included than with others of the inventory. 

■Ajialysis o f variance . Using the appraisal and reappraisal 
of the two selected articles hy the ; eight instructors of 
educational research methods , a^^'-MiOYA ^:a.s 'performed for each', / 
of the six EIPEa sections. She primary purpqse of the ahalyses- 
was to obtain reliability data from the significance tests for- 
main and interaction . effects of -.the four factors involved in 
the study (i. e. , ■ "Article", ' "Rater'', "Trait" , and "Occasion") ' 
/ and thereby to estimate- reliability co efficients on the ^ basis 
of the various variance- components. 

In all of the AlfOVA's, "Occasipn" and "Trait" were considered 
• to be fixed factors; "Rater" and "Article", random. In theory, 
however, the appropriate error term is not available . for the 
source of variance of a fj.x:ed effect variable. Accordingly, 
^ Myers (1972) suggests applying a quasi|P ratio to test the 
statistical signific'ance of such an" effect. In this study, 
Myers' technique was,, folio wed, to test the main effect of both 
"Trait" and "Occasion" as well as their associated interaction 
effects. , . . 
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Table 4 slro-ws. , the results of MOVA for the sfection of 



V 



Research/ Problem. The main effect of "Trait" was' found highly'' 

.^Insert'Ta^'ble. 4 about here 



significant at the'' .01 , level (12," 11. 81 j = 6.-56, £ <.01i. 'j 
A statistically significant "Trait" main effect indicates that V 
appraisals, averaged across' raters, articles, and occasions ' 
differed fj?^m trait to\ trait to some extent, obviously, a 
desirable/attribute for the inventory to possess,' 

In .the analysis for the section of Related Literature 
^(see Table 5),' the main effeQii of "Article" was found significant 



Insert. Table 5 about here 



J 



(F (1, 1) = 10.23, J2<.05). So/ was ! the intetractian effect 
between "Article" and "Trait" (P {3 i^) = 2.90, s < . 05) . Both 
outcomes were desirable and favorable to this section 'of EIPER 
in the sense that the appraisals differentiated article, from . 
article on the various traits presented , in' the instrument, • 
across rat^ers and.. o ccasion's as well. 

The analysis performed jZi the section of liesearch Design 
result^ed in both desirable s/nd undesirable effects. As shown 
in Table ,6, the desirable main effect of "Trait" was highly 
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significant, at the .00,1 level (29, 43- 1 3) r '5 . 35 , £V.0Q1). 
The imdesiralile oatcomes were mainly due to the various inter- 
action 'effects. V' 

Tahle 7 presents /the results of ^analysis "for the sectioi^'^ 
of Data, Coller ,,n an.d Analysis. In addition to the highly/ 

■ " - ■" r"' — - , ,• ^ / , 

Insert Tahle 7 ahout here • •/ 



significant main effect of ••Article", two interaction eff.ects~ 
were found- significant, one desirable, between . "Article^^ and ; 
•Virait''^ and the other undesir^^ between ••Eater" and "Trait" 
Tv/o slightly different pictures were ldra*h from t ^ : 

.analyses of the, last two sections of the inventory. As shown. 



in Table 8, the ••Trait" main effect 'was found 



Insert Table 8 abouit here 



statistically 



significant in the section of Concldsions and Generalizations 
(I (9, 11.12'}\.~ '3i4o, s<-01)» but nbt in ..that of Style and 
Orgariiaatioh.,as i^esented. '^ii^.-Tab;!!^ contrary, a 

significant ''Article''' ma-iji. 'e^^^^ (P (1, ''7} = 6.22, £< .05) • 
was ^ found in the analysis for the section of Style and ' 
Organization (see Table 9), hut not in that,, for Conclusion ■ 
and Generalizations. 
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*hen .he entire Inventory it W ,vere- oomMn W and analyzed 
excluding the 6 overall evaluation items . two de.irabW „ain ' 
.effects were found significant, i.e., "Article" and ■■.Iralt" ' 
(see lahle ,10,,. m addition to one first order interaction and ■ 
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...two second order interactions. On the basis of the finding, 
-from the various analyses performed, it is quite evident that 

- as a whole the inventory generated satisfactory internal . ■ 
consistency and stability-of-rasponse reliabilities. 

■ ^iaates_of^iahility indices. She reaulta of the ' ■ 

^ IK-OWs were used, to estimate three.,kinds' of reliability , 
coefficients (i.e.', interrater, .intrarater, and stability of ' 
measure of traits) .publicized by Stanley, and Wiley ' (i^g?) and 
rather, easily estimated by procedures developed and described 

• by Silverstein (I974). 

Ihe interrater reliability coefficient is the average 
correlation between the ratihgs of, an inventory item made by 

■ the eight judges, with .he average obtained' across articles 
and occasions. She intraiater reliability coefficlint' is the ■ 
average correlation between,a judge-s appraisals reannraisals 
Of the items . in an inventory section, averaged acro/s raters and- 
occasions. Ihe stability of trait, measurement rel/iability 
coefficient is the .average correlation between- th^ Judges' ' 
appraisals and reappraisals of an\lnventory item, averaged across 



articles and traits. Tatle 11 contains the estimated indi 



ces 
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for the three types of reliabilities.' 2^^^^^ 

computed for- ea'ch of the six inventory sections, :the EIPER ! 

composite, and, the "global" evaluations pertaining to the six 

major aspects of experimental research. 

Results of the "usef ulness" e. .,eriment . Por ea^i. of the 
^ six EIPER sections, the proportion; of users iii agreement with 

the "global" appraisals of the eight insiructors was compared 
.with the proportion of non-users As shown in Table 12, the 

' ■ Insert Table 1 2 about here ' ., '. ' 

, smallest of. the 1 2 proportions was ..72, with the other 11. in 
the: interval, .76 - .89. /None of the differences was statis- 
'tically significant at the ,05 significance level;, ' 

Two circumstances may account for the high proportions of 
agreement and for the non-statisticaliy significant differences. 
The comparisons were made hear the end of . the. quarter after, both 
groups, had had Identical experiences in research appraisal. 
Moreover, some of the, instructors had stressed many of the 
characteristics, contained in EIPER. Por reasons pointed out in 
the report of which this is a'stimmari^ the exT>eriment as ' 
conducted differed in important respects from the experiment 
as originally planned. 



Conclusions ■ , 'l^ 

'She goal of improving the quality of educational research . 

is not new among professionals in educational circles. Such a/, 
goal will be far from reaching or is iikely to be illusory,, 
however, unless professional concern is translated into serious 
efforts . One such effort is to develop an evaluationfi'ns'truraent : 
to assist the prospective educational researcher to appraise 
published experimental research studies critically and yet 
objectively... EI I'ER was.':developed especially for such a. purpose. 

/ , On the basis of the empirical evidence obtained in this 
.study,: the inventory has been cha-racterized with the sa-iisfactory 
psychometric properties ■■■^deeme.d to be -essential, though sdme 
improvement is desirable. ?irg„t ^ of all, the results .of item ' 
analysis gave a, clear indi cation that most inventory items were 
adequately nested in thg sections , to which they should belong. 
Ihe.n, the structure of both separate sections and the complete 
inventory was effective and favorable as explored from various 
reliability estimates via K-R 20 approach. : Furthermore, the 
desirable outcomes of main effects, from a series, of AK07A tests 
demonstrated that use of-' the inventory would, result in differ- 
ential appraisal of the two different resea'rch reports with 
s-etarate characteristics, as proved with ' satisfactory internal 
consistency and stability-pf-resnon'se, reliabilities. 

It is reasonable to /conclude that 'EIFEK in its. present 
jform is a satisfactory , aid to teaching students of research 
methods classes. Since ' the- essential characteristics of 
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;experimen1ial research are basically the same regardless of ' 
jthe di^ciRline of interest, the inventory should prove useful 
iiot only in introductory research methods classes in education 
. and psychology but in other res-earch methods classes within ' 
the spectacujliar domain of social sciences. 
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Pilot Study 




Control 




Group 




N = 55 





/ G-roup 
N = 5.8 



After One. Month 



I Second^ Evaluatioii 




Panel of 
Judges 
N= 8 



Estimate the , Selected 
Psychometric Characteristics 



Figure 1. Plow Chart of Development. Procedures of tJi^B 
• Evaluation Instrument for ^Ixperirsntal 
Research (EIPER). 
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Article 1 



Article 2 



0 

ERIC 



Rater 



Pretest Po-sttest 



Pretest Posttest 



n 



n 



n 1 . ... n 



2 
3 

5 
6 
7 
8 



1111 



1121 



1131. • * 



■11,41 • • 



1151 V • 



■1161 • • 



•1171 • • 



1181 " • 



• • * ^221n 



222n 



> Y 



223n 



\ 



. Y 



224n 



• ^225n-// 



. Y 



226n 



, Y ■ 

* >227n 

• ^22^n 



\ I. 



Figure 2. Analysis of Unreplicated A x B x G x D ^, 7 
. 4-Way Pactorical Design, with 2 Articles,/ 
8 Raters, Varia^ble Traits,, and 2 Occasions 



: ■ ~ (TABLE t . 

KK-20 Reliability Coefficients for EIPER Section: 



u^uui, 'Curitrol''^ -E>:p.-'^ -Res. Seminar^ Judges^ 
Article; 1 : 1 1 2 1 1 2 

Occasion: Pre ^"^re Pre Post Pre Post Pre Post Pre Post 

Section 

Research - " ° 

Problem , -71 .89 .94 .88 .87 . 91 .95 . 89 . 82 

(15 items) , 

Related 

Literature r .45 .7 6 • 92 .82 .7 6 i 63 . 72 . 82 .72 

( 6 items') 

iDesi^^n . .87 .94 .96 .91 .95 .88 .96 ,76 .93 

(30 items) 

•Data Col- " ' 

lection and . .84 .IQ .33. .S3 .95 .80 .76, .65 .72 

Analysis / * v 
(11 items) 

Conclusions ' / t 

and General!- . 68 . 81 .94 .90 .91 .80 .94 .65 ^.31 

stations . . . ^ v : , 

(10; items.) . ■■ — 

Style and 

Organization .72 . 90 V94 . 65 .83 .47 . 65 . 65 .52 

(3 items) • . ' 

Total , .93 .97 .99 ^96 ..98 .95 .98 .90 .91 

(73 .items). ^ ... , 

Global ^ ; "\ 

"Evaluations---^. 45. l . 64 . 81 . 97 . 88 . 93 • 84 . 90 .75 . 76 

(6-i'tems) ; \- ' T — ^ 

■ ' = 55 • 

. V\ = 58 ' 

-^n - 13 in pretest; -11 in posttest. 

^ * ■/!•"■ 
■ d_ r.. 
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TABLE 2, 

Matrix of Correlations Between EIPER Sections 





Test ■ 


.1 


2 


3 


4 


5 


6 . 


1 


Research.. 
Problem 




- 








. ''' 


.2 


Related . • ' 
, Literature 


.55 












3 


Research 
Design 


.48 


.53 










4 


Lata Collection 
and Analysis 


.56 


.55 


.69 








5 


Conclusions and 
Generalizations 


. 28 


. 47 


.54 


^ .55 






6 


Style and 
Organizatian 


.40 


. 30 


.4.1 


..50 


• 39 




7 

1 


Total* ' 


.56 ' 


.63 


.72>- 


.78 


.58 


.50 



.Corrected for overlap* 



/ 



TABLE 3 ; , •••r-., , 

Matrix, of Ito;:)-Section Correlations^ 

(W =i 50) 



rjoction 



Item 


I 


.11 


■ III 


IV 


■ V-/ 


VI 


Total 




,1.' 


. .42* 


.27 


.37 


.33' 


.30 


.38 


.44 




2 . 




.34 


.41 


• 52. 


.31 


.35 


.54 








.13 




. 22 


.23 


.22 


.25 


4 


. 37 


.10 


.26 


.36 


.39 




- .38'-; 


5 


.27 


.^2 


.23 


.30 . ' 


.21 


.23 


.32 


6 


.27 


. 32 


.11 . : 


.20 


.02 


.16 


.20 


7 


-.02 


. 08 


.08 


■.06 


-.22 


-.07 


-.01 


8 


. 20 


• 33 


'.25 


.18 


.24 


.05 


.27 


9 


.62^ 


.35 


.39 


.59 . 


.22 


i40 




10. ; 


.35 


.. 39 


.24 


.36 


.01 


.09 X 


.32 


il 


.50* 

^ ^ 


.48 


.24 


.19 


.'.15; 


.15 




12 ' 


.28 


.12 


.30 


.27 


•19 


. 18 


.32 


15 


.39 


.21 


.13 


.21 


- . 04 ' 


.16 


.21 


15 


.19 


* 

.39 


.32 / 


.28 . 


> 21 


•03 


•33 


16 


.32 


.34 


.27 


.36 


•51 


.26 ■; 


.41 


17 


.39 


.1'2 


, .25 


■.45 


. 30 


.31 


.39 


.18 


35 




. 32 


. 31. 


.26 


.22 


.37/ 


19: 


.26 ' 


.'08 


.14 


.12 


>.or' 


_.13^- 


r' ' "~ u 


20 




* 

_:4-i^ 


. 56 - - 


.30 


:T26 


707 


.38 




22 


; .10 


•39 . 


.11 ^^ 


-19 . 


.27 


.2? 


.22 


23 


• .50 


.54 


.44 


.53 


.32 


.29 


.57 


24 


.47 


.39 


. 7.1 


.44 


.41 


.41 


.46 


25 


- .25 


.. .05 




.16 


,10 


.14 


.27 


26 


•28 : 


.05 




.1 6: 


-.01 


•21 


.28 


27C 


.08 


,15. 


. 26 


.30 


,30. 


.18 


.29 


28 


.45 


, 59 


\^ : 


.40 


' .51 


.26 


, .51 


-29 


• .52 


. 39 


.44. 


, 28 


.25 


•57 



, , Table 3 --continued: 



Section / > 



Item 


I 


II 


■ III 






• VI 


Total 




S4 


— ^ — . 

■ • 't 1 


■ .55 

*4S 


.4.7 




.4.1 

• *T • 


.70 
• • 




• ;/o - 




A 6 


^ i 1 




A7 




A7 


AO 






^0 










OA 


^ Pi 
* t- 1 


OQ 








. 04. 




.'25* 

^* ^ > 


1*^ 


18 




22 ^ ' 






OA 


. 29* : 


IS 


9 A 


* 14 






- 0 fi 


OQ 


Ofi 


11 




17 


j( 


• i 0 






• ;> * 


0*^ 




■ V 


AR 


•"'■^ -PI ■ 


■ ■ 4 .6 




^R 


- "^0 






- 1 


17 


W;.-'' rzA 




- 1Q 


41 






1 R ^ ^ 






?7 




^7 




• 


• c)j 




1A 


OQ 


Oi^ 


91 






OA 




; ■ • 1 ^ 


0^ 


- 06 


99 




AR 




AQ 


t=;ft 


A A 








.28 


.46 


^* 


. 30 


.^4 ■ 


.13 

• • ✓ 


^44 








A9 


V44 


• ^0 


'^A 


' AQ 


A A 
*r O 


i 

• 10 


# c D 


AR 


• c I 




• _ 


A9 ^ 


4 f 






• 43 ' 
.51 


1 


OR 


- OA 




Aft 


10 ^ 




• y f 




- 1 Q 




^AQ 




. 1 R 


-.54* 


• *f > 


. 2Q 


-34 


1 SO 




10 




54* 
^50* 






10 


A*^ 




.16 


.25 


.30 • 


,22 


.14' 


: .42 




.32 


.21 






. 39 


• .37 


; • 50 • 


(54^- ■ 


..38 


• 22 


.59 ' 




.33 


i45 


.61 


\55 - 


.,39: 


.34 


.50 


.37 




.56 




'•21 ■ ., 


■---.♦38 ■ 


.23 




.28 


.09 


.32 


57 


.36 




.51 • 


.63*". 


.4-3 


.33 


. .60 


58 


.13 




.50 


4 ? 


.39 


^14 ' , 


.46 


59 


• ■ *32 


.•49 


. 61" 


68* 


•33 


.45 


.69 



Table 5— continued 




.globaf1l;iu^?JSsf:^^^ = (excluaing the 6 ite.s for 
e^o.^^^^^J!'^^^^^^-^^ of ,26 or • 

T-^, Research Problem; II: Related Literature- ' 

III: Research Design; IV: Data Collection and A^IWsis- 
V: Conclusions and Generalizations; VI:> style 
Organization. .„ » -h. oo^xe djm 

^"Tj^^.i^'^^^li^ing identifies the EIFER section in 
which the item was placed. . ., , "^^^lon m 

•itM.*^?^ '^J^+v,''"'''''^^''^^^ higher with the section in which 
it is p..aced than it correlates with other sections? "^^^^^ ' 
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Ai-ticles X Kaiers x 'yraity x Occasions 
' Analysis of Variance: 
BIFER Section One, Research protlera 



Source 




MS 


F 


Articles (A) 


1 j 


8.37 • 


3. 39 


Occasions (D) 


/ 


.41 


.62 


Haters (B) 


7 


1.23 


• .50 


... Traits (G) 


12 


.79 


6.56** 


- A X D 


1 


.54 


2.08 


A X B 


■ 7 


2.47 




if) X B 


7 


.38 


1.45 


A X C 


12 


.09 


. 60' 


D X 0 


1 


• C.C. 




B X C 


84 


.18 


1.24 


A x'^D X B 


7 


.26 




A X D X C 


12 


.11 


1.59 


A X B X C 


84 


.14 




D X B X C 


84 


.07 


1,01 


., A X 3 X 0 X D 


84^:' 


.07- 





Note. Numlier of items = 13. 



ERIC 



'iABLS 5 



Articles x' Rate: 



X Traiin x Occo.sions 



/jialy y i 3 of Var j. tin o e : 
EIFER Section Swo, Related Literature 



Source 


df 


j Iv;3 




1 ■ 


7.b2 




1 


. 08 


iiat ers \^i3 j , 


7 


1.26 




5 


.36 


A X D 


1 


. .02 


A T*> 

A X B , 


7 


.74 


TS V "D 

iJ X ±5 


7 


.04 


A X jL/ . 


5 


• 57 ■ 


D X C 


5 


.08 ; 


B X C 


35 


.23 


A X D X B 


7 


.02 


A X D X C 


5 . 


.. . 10 


A X B X C 


35 


.13 


D X B X C 


35 


.09 


A X B X C X D , j 


35 


■ .12 



10.23 
1.93 
1 .72 
.76 
1 .00 

1.71 . 
2.90* 
1. 30 
1.80 

.77 

.75 



Note . Number of items = 6. 



£ < . 05 . 



TABLE .6. 



Arti.cles x Ratei*s x Traits x Occasions 
.Analysis of Variance:. 
EIPER Section Three, Research Design 



W Li JL ^ C 






F 


Articles (A) 


1 


p. 25 


3.53 


Occasion*^ T T)^ 


I 


.01 


.01 


Raters fB) 


( 


4« 57 


3.14 


Traits (C) 






5. 35 


A X D ' 


1 


• 30 


6.86 ,t- 


. A X B 


7 






D X B 


7 


• o3 


1 8* 87 


A X C 






1 . 08 


D X C 


29 


. 20 ■ 


1.88 


' B X C 


203 


.24 


/ -1.47^* . 


A X D X B^^^ 


7 


.04 




A X B X C 


29 


.10 

I,. 


.96^ 


A X Bx C 


203. 






I) X B X C 


203 


.11 


1.05 


A X B X C. X B 


203 

- 


- .10 . 




Note. •Nura'ber of items = 30. 
*P<.05. 


1 



7 



Articles x Haters x Traits x Occasions 
* Analysis of Variance: 
EIPJER Section Four, Data Collection and Analysis 



Soizrc e 


df 


MS 


F 


Articles (A) 


1 


5.50 


12.28** 


Occasions (D) 


1 


.18 


. 30 


Raters (B) 


7 


1.14 


2.55 


Traits (G) 


10 


.91 


2,11 


A. X D 


1 


.92 


1.87 


" A.x 3 T- 


7 


.45 




I> X B 


7 


.18 


. 37 


A X 0 


10 


.31 


2.37 


D X C . 


10 


.09 


1.00 


B X C 


70 


.25 


1.88**" 


A X D X B 


■ 7 


.49 




Ax D X C 


10 


.07, 


.71 


A X B x C 


. 70 


.15 




D X B X G 


70 


.12 


T,. 24 


A X B X C X D 


70 


.10 





Note . Number of. items -= 1 1 . 



*p <.05. 
.p<.01. 

■ ■ / 
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ICAB.liS 8 ■ 

Articles x Jiators x j?rri.ltG x OccciGiono 
Arinlyr^is oJ; Vnrinnoo: 
EIFER Section Pivc^ 'Concluoior:i3 and Generalizations 



Source 


dl 1 MS 




Articles (A) 




5.00 • 


2.72 


Occasiono (D) 


1 


.00 


.01 


Haters (B) 


7 


1..4 6 


1.32 ' 


Traits (C) ■ 


9-. 


1.02 


5.20**1 


A X D 


1 


.25 


1 . 82 


A X B 


- 7 


1.10 




D X B ■ 


7 


.50 


2.18 


A X C 


9 




.97 


. D X C 


- 9 


? 19 


.92 


B X C 


63 


.20 


1 . 31 


A X D X B 




.14 




A X D X C / 


9 


.22 


1 . 61 


A X B X C 


63 


.15 




D X B X C 


63 


.13 


.92 


A X. B X C X D ' ' 


63 


■ ' ,14 





Note . Number of items = 10. .;' 

•* - - ■- ■ ' 

p <. 01 . : ■ ' ■ 



32 



Ar-iclos >: Katerij x 'Iraits x Occasions 
Analysis of Variance: 
EIFDK Section Six, Style and Crganization 



Source 




MS 




Articles (A) 


1 


2 . 67 


6,22* . 


Occasions (D) 


1 


.67 


2.80 


Raters (B; 


• 7 


.55 


r.28 


iraits {^C; 


2 


•59 


3.59 


A X 2) 


1 


.17. 


1,40 


A X B 


7 


.43 




D X B 


7 


.19 


1 , 60 


A X C 


2 


.07 


.86 / 


D X C 


2 


.14 


.32 


B X C 


14 


.18 • 


2.09 


A X P X 


7 


,12 




>A. X D X C 


.2 


•^5 


2.98 


A X B X C 


. 14 


.09 




D X B X C 


14 


.12 


.82 


A?cBxCxD 
. . . . 


14 


.15 





-Note. Number of items = 3. 
<.05. 




TA13LE 10 

Articles x Katerr, x Traitn x Occn.-.lon.s 
_Aiialynir. oX Variajico; 
^.^EIPJDR Section Seven, 'I'utal Te.i:t 

\ 

" • ' ' \, . -. ■ . 



Source 


— 




— — ~ _ 

iV 




1 


20.72 


6.75* 


0 C Crist* on <^ r 


1 


.01 


,03 




7 


6.95 


1 . 63 


ffrai (OA 


72 


1 . 1 1 


V , V- .U. 

3.62* 


A JJ 


1 


.02 , 


.06 


A X 


7 


4.26 




T\ Tt 
JJ X J3 


7 , 


.74 


2. 14 . 


^ »«v 


72 


■ -..53- 


1.31 


D X C 


.72 


.18 


1.18 


B X C 


504 


.25 


1.47** 


A x D X B 


7 


.35 




A X D X C 


72 


.15 


1.36* 


Ax B X C 


504 


.17 




. 2 X B X C 


504 


■ .11 


1.G7* 


X 3 X C X D 


504 


.11 





S^te. Number of items = 73, 



^ < .05. 

'"'^ <, 01 . 
*^Ti^<. ..DDI . 
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-t«-.o'=/-uxxexau„on i^e-^ween the 



liaters, I- raits,, and Dccasl 



Iiitsrrat er 



Section 



.^lesearch prolDlem 
•Related-Lit erature 
■ Research Design 

Data Collection 
and Analysis 

Conclusions and 
Generalizations 

Style and 
Oarganization 

Total test 

G-lol3al Braluations 



V -, . ^...^ ' Intrarater Stabklit\r 
Kel.ability Keliability of Traits 



.12 

. 33. 
.06 

.21 

.08 

.26 
.14 
.16 



. 60 

.38 
.27 

. 30 

. 28 



- • 4 4-. A 
. 18 

. 33 
.29 

.41 

.26 

■ .57 
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SABLE 12 



Proportion of Responces i?o Each of the 
Si3c Overall L"val\iation Items* 



I. Research Pro ol era 
0 1 



II. Related Lit eriature 
0 1 



E 


9 

(.15) 


49 
(.35) 


58. 




■13 
(.22) 


45 
(.78) 


58 


C 


8 

(.15) 


47 

(.85) 


55 


C 


13 
(.24) 


42 
(.76) 


55 




- ^2 = 


96 
. 0 67 , 2 


= 0 


/ 

s 

1^1 

// 


2 6 
- P2 = 


87 

-079, z 


= -.25 


Res 


earch MBsign 
0 1 




IV. Data Gollec. and Aniily.' 

.. ./ 0 1 


E 


16 

^(.2B) 


(.72) 


58 


S 


9 

(.16) 


49 
(.84) 


58 


C 


11 
(.20) 


44 
■(.80) 


55 


c 


6 

(.11) 


49 
(.89) 


55 




27 
- P2 = 


86 
.08, z 


= 1 


/■ - 


15 
- P2 = 


98 
.063, 2 


= -.79 



y. Conclu. and General. 



E 
C 

s_ 



0 


1 


1 3 


45 


(.22) 


(.78) 


9 


46' 


(.16) 




22 


91 


- .^2 = 


.074, z 



58 

35\ 



VI. Style and Organization 

* ' 0 ^ 1 

5B 



E 



10 


48 


(^17) 


(.55) 


6 


49' 


(.11) 


(.89) ■ 


16 


. 97 


" P2 = 


.065, 2 



2 = .92 



I^ch item corresponds /to the section of the 
inventory.; /,^ 
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<7 
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EVALUATION OF RESEARCH REPORT 



Yi-Ming Hsu 
West Chester State. College 



Instructions 

The attached inventory is being formulated as a guide in evaluating; repor 
ted experimental research in education and psychology. -It was developed fol- 
lowing an extens^ive review of literature to identify attributes in the re- 
search process which characterize experimental research. Each statement has 
been checked by nationalfly known experts in educational research methodology 
as essential to experimental^ research. In addition, in the judgment of pro- 
fessors of educational research across the country, students in the intro- 
ductory course of research methods should be able to recognize and appraise 
the attributes as they are presented in published reports of experimental 
research. ' t 

: The statements aire organized u^der 'th^ following major headings to facili 
tate the rater in making evaluations: 

■ . ■ ' • 

I . RESEARCH PROBLEM M 

A. Protlem statement ' 

B. Hypothesis(es) 

i;, RELATED LITERATURE ^ . . . , 

III. RESEARCH DESIGN . ^ -/ 

. A. ' The population and sample , 

B. The experimental arrangements . 

C. Controls, for the possible threats zo the internal validity 

D. Controls for the possible threats to the external, validity . 

IV. ^ DATA COLLECTION AND ANALYSIS 

A. Data collection ^ . 

B. Data analysis 

V. CONCLUSIONS AND GENERALIZATIONS ^ • ; 

VI.. STYLE AND ORGANIZATION OF THE REt>ORT 

To make a proper use of the inventory, T\^0 different categories of eval-- 
uations are required: 

(i) An evaluation of the specific statement of the attribute^, 
(ii) An overall or *'global" evaluation of the aspect of the re- 
search process. . ^ 



.^Copyrighted 1975. Not ^for reproduction or use without the permission of 
the. author. 



In judging ,the research with respect to each ^sgocif Ic attribute, please 
use the followilng key: 

1: The article -contains no information concerning the attribute. 
2: The information given clearly indicates that the attribute was 

lnapprop>riately handled. 
'3: The information gt^en suggests that the attribute inay have 

been inp roper ly managed. 
4: The information given; indicates that the attribute was properly 

managed. \ . 

5: The infoiimatlon given clearly i:idicates that the attribute was 
appropriately handled., 

The overall evaluation at the end of each section should NOT be deter- 
mined by adding and/or averaging the responses'* to the separate statements. 
Instead, please make^^an overall or "global" appraisal of each aspect of the 
research using the same key and Instructions (replacing the word '^attribute" 
with '^aspect^O as you respond to the ' specif ic items. 

The order in which che attributes are -listed on the inventory will prob- 
ably not be sa;ne as the order in which they are identified in the published 
reports of experimental research. For this reason you should follow the pro 
cedures described helow. so as to produce more dependable evaluations: 

A. Read the statements on the inventory carefully* 
R. Read the research report' in its entirety without 
a"t,tejnhting to evaluate it.. 

C. Reread" the report searching information relevant 
to the separate items on the inventory. 

D. Refer to the key as often as needed in making 
your* evaluation. 

E. Mark the appropriate responses, on the answer 
sheet provided (be sure to. use -#2 pencil). 

F. Check the answer sheet to see that you have com- 
pleted both each sp ecific item evaluation and the , 

SIX overall evaluations, and that. you have vrltten i 
the information asked for. , 



, • EVALUATION INSTRUMENT FOR EXPERIMENTAL RESEARCH (EIFER)^ 

,Yi-Ming Hsu 
West Chester State College 

I. RESEARCH PROBLEM 

(i) Specific Items 

Problem statement 

1. The research problem is clearly stated and precisely defined/ 

2. The significance of the study is demonstrated. 

3. The relationship of the study to its scientific or exper.i-^ 
ential antecedents is indicated, ; / 

4. The objectives of the study are described. 
' \ 5. Assumptions of the study ar^ stated. 

6. Limitations of the study are noted. p 

7. Critical: or unusual terms are defined. j 

B. Hypothesis (es) • - • . | 

8. The .hy5>othesis(es) is(are) easily identified*' 

w 9. The hyjothesis(es) is (are) derived from the research problem. 

10. T^e logical and empirical framework frpmj which the hypothe- 
' ^' ' .sis(es) was (were) derived is. demonstrated. 

11. The liypothesis(es) clearly id'entifies(fyj the independent 
variables ^; . I 

12. The hypothesis (es) clearly identif ies(fy)\ the effects to. 
be measured by the dependent variable (s) . . ' / 

13. The h3rpotfiesis(es) is(are) testable. 

(ii) Overall or "global" evaluation 
. ' 14. RESEARCH PROBLEM 
II. RELATED 'LITERAIlJRE 
(i) , Specific items 

15. Literature review is thorough and comprehensive. ' 

16. ^Literature review is well-organized. 

17. Literature reviewed is directly relevant to the research study 
, 18; The theoretical basis for the problem is identified. 

19. The methodological strengths and/or weaknesses of the study 
are discussed. ^ . * 

20. The^research design accounts for the variables which have prob- 
*^bly influences on the dependent variable (s). 

. .... ■■ ■ • . • ■ ' ' / • . , ■ ■ ■ ■ 

(ii) Overall or "global** evaluation 

' 21. RELATED LITERATURE 
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III. RESEARCH DESIGN 

(i) Specific items 



A. The population and sample 

22. The population to which generalization will be made is 
clearly specified. 

23. The characteristics of the sample are fully described. 

24. The sample size in the study is indicated. 

25. Procedures for sample selection are fully described, / , 

26. The method of assigning subjects to the comparison 
groups is clearly described. 

B. The experimental arrangements 

27. The treatment (s) is (are) randomly aissigned to the com- 
parison groups, 

28. The research design includes the independent variables 
identified in the 'hypothesis(es) . 

29. The dependent variable (s) appropriately measures the 
effect(8) identified in the hypothesisXes) . 

30. The treatment(s) is(are) sufficiently described so 
that replication of the study iriay be possible. 

31. Adequate information regarding the administration 
of the treatment (s) is provided » 

32. The treatment(s) is(are) effectively applied in accor- 
dance with the objectives of the study. 

C. The' following possible threats tb the internal validity of 
the experiment are controlled: 

33. Histoiry: Specific events, external to^the tryeatment 
CQnditiona, occurring^ during the experimentation. 

34. Maturation: Changes within the subject asi a function 

of the passage of time during the course of experiment. 

35. Testing: Variation betwee^n pretest and posttest responses 
due to cues from the pretest. 

36. Instrumentsrtloir:' Changes in the calibration of a measuring 
-Inrffffument or inconsistency of the scorers or raters can 

affect the measurements/ 

37. Statistical regression: Regression toward mean may occur 
if some but riot all subjects are sampled from extreme 
groups. 

38. Sample selection: Biases r«sulting from differences 
in the selection of subjects in the comparison groups- 

39. /Experimental mortality: The differential loss of 
subjects from the cbmparison groups during an experi- 
men t . - 

40. Interaction of Selection and maturation, etc.: An inter- 
action between selection and any other factors above which 
may be mistaken for the experimental effect. 
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D. The following possible .threats to the external validity of 
the experiment are controlled: 

(a) Population validity 

41. Accessible vs. target population: -Representativeness 
of the sample with respe;ct to the population to which 
generallzatipns are. made. 

42. Interaction of personologlcal variables and treatment 
effects: Reaction of the subjects with different per- 
sonality characteristics to the treatment \conditions. 

(b) Ecological validity ' \ 

43. Explicit definition of the independent variable: Descrip-- 
tions of the management and operation of th^ treatment(s) 
(independent variables). 1 

44. Multiple treatment inter ference : Interference with exper- 
imental results occurring from two or more treatments 
having been administered consecutively to the same 
subjects within. a. given time, period. 

45. Hawthorne ef feet :' Awareness of the experiment may affect 
the response of the subject to the experimental stimuli. 

46. Novelty and disruption effects: The experimental results 
may be due partly to the enthusiasm or disruption 
generated by the newness of the treatment. 

47. Blxperimenter effect: Certain characteristics or behaviors 
of the experimenter may unintentionally influence the . 
response of the subject. 

48. Pretest sensitization The administration of a pretest 
may have possible influences on the treatment effects. 

49. Posttest sensitization: A test following the experi- 
ment may elicit effects which otherwise would remain 
latent or incomplete. 

i 50. Interaction of history and treatment effects: The 

experimental results may be unique because of **extra- 
neous*' events occurring dur.ing. the course of the experl-t 
J ment . 

51 . Interaction of time of measurement and treatment effects: 
Measurement of the dependent variable at two different 
points of time may produce two dif f erent resu3 ts • 

(ii) Overall or "global** evaluation 

52. RESEARCH DESIGN ' 
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IV. DATA COLLECTION AND ANALYSIS 

(i) Specific items 

■ .. ■ '',./■ 

A. Data collection 

53. The rationale for selection or development of the dependent 
variable measureCs) is clearly stated. 

54. The measurement procedures adopted in the study for data 
gathering are specified. 

55. Reliability data for the effects measurements are reported. 

56. Validity data for the effects measurements are cited. v 

57. The procedures for data collection are carefully planned. 

58. Deviations, if any,^ from that plan are made explicit. 

B. Data analysis 

59.. The methods of data analysis are specifically described. 

60. The methods of data analysis are appropriate for the 
specified research design. 

61, The pattern of statistical analysis is applied ^aorrectly 
with respect to the nature of the raw data^^---' 

62, Statistical techniques are appropriate to'^he number of 
treatment groups and hypothesis (es) under consideration. 

63. The level of significance in hypothesis testing is speci- 
fied and adequate for the investigation- 



(il). Overall or "global" evaluation 

6A. DATA COLLECTION AND ANALYSIS 
^CONCLUSIONS AND GENERALIZATIONS 
(i) Specific items 

65. Tables and figures display the data basic to testing the 
hypothesis (es). 

66. Results of hypothesis testing are reported with support 
of statistical evidence. 

67. Claims for the probable truth or falsity , of the research 
hypothesis Ces) are supported by the evidence presented. 

68. Conclusions are objectively stated and effectively sumr 
. ' marized . ^ 

69. Discussions are consistent with the results presented. 

70. The extent to which the study can be generalized to the 
population of interest is clearly identified. 

71. Generalizations made are reasonable and logical. 

^72. Evidence is presented in support of the internal validity 
of the study. ' 

73. The findings are related to the previous research on the 
problem of inquiry. 

74. Problems raised from the study are stated for further 
exploration. 

(ii) Overall or "global" evaluation 

75. CONCLUSIONS AND GENERALIZATIONS 
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VI. STYLE AND ORGANIZATION OF THE REPORT 
(i) Specific items 



76 



The report is 



written in clear, underatandable language. 

77. Organization of the content is clear /and rigorous . j 

78. The style and! tone of the report re/lect an objective, unbi- 
ased, and scientific attitude. / J 



(ii) Overall or . "global'!* evaluation 



79.' STYLE AND OKGANIZATION OF THE REPORT 



/ 
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